Skip to content

Fixing test for pii operator #430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Parent: Forecasting Operator
merged 11 commits into from
Nov 16, 2023
Merged

Fixing test for pii operator #430

merged 11 commits into from
Nov 16, 2023

Conversation

@mingkang111
Copy link
Member

@mingkang111 mingkang111 commented Nov 16, 2023 •
edited
Loading

Description

This PR aims to fix the failing unit test for pii operator. It also make some improvement on documentation.

Issue:

Error happened when constructing Spacy Detector

E           ValueError: [E002] Can't find factory for 'curated_transformer' for language English (en). This usually happens when spaCy calls `nlp.create_pipe` with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator `@Language.component` (for function components) or `@Language.factory` (for class components).


FAILED tests/unitary/with_extras/operator/pii/test_factory.py::TestPiiDetectorFactory::test_get_spacy_detector[spacy-person-en_core_web_trf] - ValueError: [E002] Can't find factory for 'curated_transformer' for language English (en). This usually happens when spaCy calls `nlp.create_pipe` wit...

Try with other model, only en_core_web_trf has the problem.

Solution:

This is known issue: LeapBeyond/scrubadub_spacy#4, a workaround can be fixing the version of some dependencies.
Fixed the version for the following dependencies:
"scrubadub==2.0.1" "spacy==3.6.1" "spacy-transformers==1.2.5"

mrDzurb and others added 11 commits October 27, 2023 10:59
@mrDzurb
PII operator backbone.
8eebe45
@mingkang111
Merge branch 'feature/forecasting' of https://github.com/oracle/accel…
6b13982 
…erated-data-science into feature/pii
@mingkang111
Merge branch 'main' of https://github.com/oracle/accelerated-data-sci…
cfe002b 
…ence into feature/pii
@mrDzurb
Merge branch 'feature/forecasting' of https://github.com/oracle/accel…
09099fe 
…erated-data-science into feature/pii
@mingkang111
feature/ads_pii_operator (#424)
3677198
@mingkang111
updated pii operator's dependency
16c6222
@mingkang111
fixed docs
c45427d
@mingkang111
fixed tests
893656c
@mingkang111
Merge branch 'feature/forecasting' of https://github.com/oracle/accel…
b7a8dff 
…erated-data-science into feature/pii
@mingkang111
improved documentation for configuring detectors
52280d1
@mingkang111
fixed ValueError: [E002] Can't find factory for 'curated_transformer'…
2929f8b 
… for language English (en). for loading en_core_web_trf
@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Nov 16, 2023
@github-actions
Copy link

⚠️ This PR changed pyproject.toml file. ⚠️

  • PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
  • PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

mingkang111
mingkang111 commented Nov 16, 2023
View reviewed changes
tests/unitary/with_extras/operator/pii/test_factory.py
("spacy", "other", "en_core_web_trf"),
("spacy", "person", "en_core_web_sm"),
("spacy", "other", "en_core_web_sm"),
# ("spacy", "org", "en_core_web_trf"),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disable testing with other model because of the size. Verified successfully in local.
Screenshot 2023-11-15 at 6 27 01 PM

@mingkang111 mingkang111 changed the title Fixed test for pii operator Fixing test for pii operator Nov 16, 2023
@mingkang111 mingkang111 requested a review from mrDzurb November 16, 2023 02:28
@mingkang111 mingkang111 merged commit 6899cbc into feature/forecasting Nov 16, 2023
@github-actions
Copy link

📌 Cov diff with main:

Coverage-1%

📌 Overall coverage:

Coverage-24.30%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrDzurb mrDzurb Awaiting requested review from mrDzurb

Assignees

No one assigned

Labels

OCA Verified All contributors have signed the Oracle Contributor Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mingkang111 @mrDzurb